Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VL][Core] SampleExec Operator Native Support #5856

Merged
merged 3 commits into from
May 28, 2024
Merged

[VL][Core] SampleExec Operator Native Support #5856

merged 3 commits into from
May 28, 2024

Conversation

gaoyangxiaozhu
Copy link
Contributor

@gaoyangxiaozhu gaoyangxiaozhu commented May 23, 2024 •
edited
Loading

What changes were proposed in this pull request?

The PR introduce the SampleExec native support (SampleExecTransformer) for velox backend, basically it transfer the SampleExec spark physical operator to velox ProjectFilter operator with rand(seed) < fraction expression.

The feature is under feature flag spark.gluten.sql.columnarSampleEnabled with default is false now.

SampleExec support two sample way

  1. Sampling with replacement (withReplacement is set to true)
  2. sampling without replacement (default)

Right now, we only support for samling without replacement.

(Please fill in changes proposed in this fix)

(Fixes: #5315)

How was this patch tested?

locally test with turning on feature flag

A case use df.sample(0.5, 10)

image

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

@github-actions GitHub Actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@gaoyangxiaozhu gaoyangxiaozhu changed the title [VL][Core][WIP] SampleExec Operator Native Support [VL][Core] SampleExec Operator Native Support May 23, 2024
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@gaoyangxiaozhu
Copy link
Contributor Author

@rui-mo / @zhouyuan / @zhli1142015 help review

@FelixYBW
Copy link
Contributor

Right now, we only support for samling without replaceme

Will Sampling with replacement (withReplacement is set to true) fallback? how complex if we add support to Velox? how useful it is?

@gaoyangxiaozhu
Copy link
Contributor Author

gaoyangxiaozhu commented May 25, 2024 •
edited
Loading

Right now, we only support for samling without replaceme

Will Sampling with replacement (withReplacement is set to true) fallback? how complex if we add support to Velox? how useful it is?

sample with relpacement allow pick up same row multiple times, that means it actually not simple filter but re-generating the data set, that is not worked by simple convert to a filter rand expression. I don't yet figur out a good way to supprot that part unless to make a new sampleNode in Velox side.

about how useful withReplacement , the default is false means always sample without replacement, i think for big data analysis that's enough, the sampleWithReplacement is more suitable for small size data.

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@gaoyangxiaozhu gaoyangxiaozhu requested a review from rui-mo May 27, 2024 06:29
rui-mo
rui-mo approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@gaoyangxiaozhu
Copy link
Contributor Author

gaoyangxiaozhu commented May 28, 2024 •
edited
Loading

@zhli1142015 / @zhouyuan help merge

@zhli1142015 zhli1142015 merged commit 729d345 into apache:main May 28, 2024
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rui-mo rui-mo rui-mo approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[VL] Add SampleExec operator support
4 participants
@gaoyangxiaozhu @FelixYBW @rui-mo @zhli1142015